A soft thumb-sized vision-based sensor with accurate all-round force perception

Abstract Vision-based haptic sensors have emerged as a promising approach to robotic touch due affordable high-resolution cameras and successful computer vision techniques; however, their physical design the information they provide do not yet meet requirements of real applications. We present robust, soft, low-cost, vision-based, thumb-sized three-dimensional sensor named Insight, which continually provides directional force-distribution map over its entire conical sensing surface. Constructed around an internal monocular camera, has only single layer elastomer over-moulded on stiff frame guarantee sensitivity, robustness soft contact. Furthermore, Insight uniquely combines photometric stereo structured light using collimator detect deformation easily replaceable flexible outer shell. The force is inferred by deep neural network that maps images spatial distribution contact (normal shear). overall resolution 0.4 mm, magnitude accuracy 0.03 N direction five degrees range 0.03–2 for numerous distinct contacts with varying area. presented hardware software concepts can be transferred wide variety robot parts.